Method and device of processing plaques in magnetic resonance imaging of vessel wall, and computer device

ABSTRACT

A method of processing plaques in magnetic resonance imaging of vessel wall include: step S 101,  training a generative adversarial network and a capsule neural network to obtain a trained generator network and a trained capsule neural network; and step S 102,  cascade-connecting the trained generator network with the capsule neural network into a system to recognize and classify plaques in magnetic resonance imaging of vessel wall. In one aspect, the capsule neural network has more abundant vascular plaques characteristic information represented by vector; in another aspect, when the trained generator network and the capsule neural network are cascaded into the system to recognize and classify the plaques in magnetic resonance imaging of vessel wall, an accuracy of recognition and classification may be greatly improved. A device for processing the method as well as a computer for implementing are also disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation-in-Part of PCT/CN2019/078890,filed on Mar. 20, 2019, which claims priority to Chinese patentapplication No. 201810818827.1, filed on Jul. 24, 2018 and entitled“Method and Device of Treating Plaque From Magnetic Resonance VascularWall Imaging,” the entire disclosures of which are incorporated hereinby reference in their entireties.

BACKGROUND 1. Technical Field

The present disclosure relates to the technical field of imageprocessing, and particularly to a method of processing plaques inmagnetic resonance imaging of vessel wall, a device of processingplaques in magnetic resonance imaging of vessel wall, and a computerdevice.

2. Related Art

Magnetic resonance imaging of vessel wall may not only performquantitative analysis on vascular plaques of the whole body such asintracranial artery, carotid artery and aorta, but also recognizeinstability characteristics such as fiber cap of vulnerable plaques,bleeding, calcification, lipid nucleus, inflammation and the likeaccurately, it is the currently recognized best plaques imaging method.However, since data volume of three-dimensional high-resolution magneticresonance imaging of vessel wall is huge, the number of images of eachinspector may reach 500, so that an experienced professional needs totake 30 minutes to complete the diagnosis of one inspector.

The deep learning method applied in the medical field is a convolutionalneural network, and the application field of the convolutional neuralnetwork includes medical image treatment, medical image recognition andclassification, and the like. Researches of deep learning in medicalimage recognition and segmentation are mostly based on traditionalconvolutional neural network algorithms. However, image details may notbe excellently processed by the deep learning method based on theconvolutional neural network due to loss of information. In theclassification problem, the convolutional and full-connectivity-basedclassification network have developed into a mature network structure,however, the accuracy of classification is still not high.

As described above, there exists a deficiency of low efficiency and lowaccuracy in recognition and classification of plaques in magneticresonance imaging of vessel wall in the existing methods.

SUMMARY

A purpose of the present disclosure is providing a method and device ofprocessing plaques in magnetic resonance imaging of vessel wall, and acomputer device to recognize and classify plaques in magnetic resonanceimaging of vessel wall high efficiently and accurately.

In a first aspect, embodiments of the present disclosure provide amethod of processing plaques in magnetic resonance imaging of vesselwall implemented by a computer device, including:

training, by the computer device, a generative adversarial network and acapsule neural network so as to obtain a trained generator network and atrained capsule neural network; and

cascade-connecting, by the computer device, the trained generatornetwork with the capsule neural network into a system, to recognize andclassify the plaques in magnetic resonance imaging of vessel wall.

In a second aspect, embodiments of the present disclosure provide anetwork of processing plaques in magnetic resonance imaging of vesselwall, including a generative adversarial network and a capsule neuralnetwork;

where the generative adversarial network is configured to recognizeplaques in magnetic resonance imaging of vessel wall after beingtrained, and the capsule neural network is configured to classify theplaques in magnetic resonance imaging of vessel wall.

In a third aspect, embodiments of the present disclosure provide acomputing device, including a memory, a processor and a computer programstored in the memory and executable on the processor, where whenexecuting the computer program, the processor is configured to implementsteps in the method described as follows:

training a generative adversarial network and a capsule neural networkso as to obtain a trained generator network and a trained capsule neuralnetwork; and

cascade-connecting the trained generator network with the capsule neuralnetwork into a system, to recognize and classify the plaques in magneticresonance imaging of vessel wall.

In a fourth aspect, embodiments of the present disclosure provide acomputer readable storage medium which stores a computer program, wherewhen the computer program is executed by a processor, steps of traininga generative adversarial network and a capsule neural network so as toobtain a trained generator network and a trained capsule neural network,and cascade-connecting the trained generator network with the capsuleneural network into a system so as to recognize and classify the plaquesin magnetic resonance imaging of vessel wall in the method areimplemented.

It may be seen from the technical solutions of the present disclosurethat, in one aspect, since the generative adversarial network includes acapsule neural network, as compared to the traditional convolutionalneural network or a fully-connected layer neural network which usesscalar to represent the vascular plaques, the capsule neural network hasmore abundant vascular plaques characteristic information represented bythe vector; in another aspect, as compared to the traditional deeplearning algorithm which adopts a gradient propagation method, thecapsule neural network uses the dynamic routing algorithm to learn andupdate the network, thus, the accuracy of recognition and classificationof the plaques may be greatly improved when the trained generatornetwork and the capsule neural network are cascaded into a system torecognize and classify the plaques in magnetic resonance imaging ofvessel wall.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the invention will be betterunderstood when considered in connection with the following detaileddescription and drawings, in which:

FIG. 1 illustrates a schematic flowchart of implementation of a methodof processing plaques in magnetic resonance imaging of vessel wallprovided by one embodiment of the present disclosure; and

FIG. 2 illustrates a schematic structural block diagram of a computerdevice provided by one embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In order to make the purpose, the technical solution and theadvantageous effects of the present disclosure be clearer and moreunderstandable, the present disclosure will be further described indetail below with reference to accompanying figures and embodiments. Itshould be understood that the specific embodiments described herein aremerely intended to illustrate but not to limit the present disclosure.

In the following description, in order to describe but not intended tolimit, concrete details such as specific system structure, technique andso on are proposed, so that comprehensive understanding of theembodiments of the present disclosure is facilitated. However, it willbe apparent to the ordinarily skilled one in the art that, the presentdisclosure may also be implemented in some other embodiments withoutthese concrete details. In some other conditions, detailed explanationsof method, circuit, device and system well known to the public areomitted, so that unnecessary details may be prevented from obstructingthe description of the present disclosure.

FIG. 1 is a schematic flowchart of implementation of a method ofprocessing plaques in magnetic resonance imaging of vessel wallaccording to an embodiment of the present disclosure, the method mainlyincludes the following step S101 and step S102, which are described indetail below:

In a step of S101, training a generative adversarial network and acapsule neural network so as to obtain a trained generator network and atrained capsule neural network.

In this embodiment of the present disclosure, the generative adversarialnetwork includes a discriminator network and a generator network, wherethe discriminator network applies a hybrid structure of conventionalneural network (e.g., convolutional neural network, etc.) and thecapsule neural network, and includes a convolutional layer, aPrimaryCaps layer and a DigitCaps layer, the generator network is a deepconvolutional network, and the generator network adopts a residualnetwork structure in consideration for more effectively training thedeep network; a parametric rectified linear unit may be used as anactivation function in the discriminator network and the generatornetwork. The capsule neural network is similar to the discriminatornetwork, the capsule neural network also includes a convolutional layer,a PrimaryCaps layer, and a DigitCaps layer, where the PrimaryCaps layeris computationally equivalent to the conventional convolutional layer,but differs from the conventional convolutional layer in deep sense,this is because the internal part of each capsule in the PrimaryCapslayer is consisted of a plurality of feature vectors, a capsule neuronof the capsule neural network uses a squashing function as theactivation function and a dynamic routing updating algorithm is used fortraining the capsule neuron, however, the convolutional neuron uses arectified linear unit as the activation function and uses a Adamalgorithm to train the conventional convolutional neuron; where thesquashing function is expressed as follows:

${\left( I^{G} \right)v_{j}} = {\frac{{s_{j}}^{2}}{1 + {s_{j}}^{2}}\frac{s_{j}}{s_{j}}}$

Where s_(j) is the total input of the capsule neural network, and itscomputational formula is expressed as follows:

$s_{j} = {\sum\limits_{i}{c_{ij}{\overset{\hat{}}{u}}_{j|i}}}$

Where parameter c_(ij) is updated by dynamic routing algorithm, û_(j|i)is the information transmitted from the ith capsule neuron to the jthcapsule neuron in the following layer, and the computational formula ofû_(j|i) is expressed as follows:û _(j|i) =W _(ij) u _(i)

Where parameter W_(ij) is obtained by the dynamic routing updatingalgorithm by learning, u_(i) is the original output of the capsuleneuron in the upper layer.

In order to make the generator network and the capsule neural networkobtained after training to be higher in recognition rate and precision,and to reduce computational resources as much as possiblesimultaneously, as one embodiment of the present disclosure, training agenerative adversarial network and a capsule neural network to obtain atrained generator network and a trained capsule neural network may beimplemented by taking three-dimensional local magnetic resonance imagingof vessel wall as training data, and using the Adam training algorithmand the dynamic routing updating algorithm to generate the adversarialnetwork and the capsule neural network so as to obtain the trainedgenerator network and the trained capsule neural network.

It should be noted that, in order to satisfy K-Lipschitz assumption ofWasserstein distance, this project uses a gradient penalty to satisfythe Lipschitz condition, and the loss function used for training thediscriminator network during training of the generative adversarialnetwork and the capsule neural network may beL(θ_(D))=E_({tilde over (x)}˜P) _(g) [D_(θ) _(D) ({tilde over(x)})]−E_(x˜P) _(r) [D_(θ) _(D) (x)]+λE_({circumflex over (x)}˜P)_({circumflex over (x)}) [(∥∇_({circumflex over (x)})D_(θ) _(D) ({tildeover (x)})∥₂−1)²], the loss function used for training the generatornetwork may be expressed as L(θ_(G))=L_(MSE)(I^(L),I^(G))+10⁻⁴L_(WANG)(I^(G)), where D_(θ) _(D) is the discriminator ofparameter θ_(D), x is magnetic resonance imaging of vessel wall data,{tilde over (x)} is the data obtained after the magnetic resonanceimaging data of vessel wall is segmented by the generator network,{circumflex over (x)}=εx+(1−ε){tilde over (x)} is a random sample ofrandom obedience distribution, ε is a random value of uniformdistribution ranged from 0 to 1, L_(MSE)(I^(L), I^(G)) is calculatedaccording to the formula of

${{L_{MSE}\left( {I^{L},I^{G}} \right)} = {\frac{1}{w \times h}{\sum\limits_{x = 1}^{w}{\sum\limits_{y = 1}^{h}\left( {I_{x,y}^{L} - I_{x,y}^{G}} \right)^{2}}}}},$L_(WANG)(I^(G)) is calculated according to the formula ofL_(WANG)(θ_(G))=−E_({tilde over (x)}˜P) _(g) [D_(θ) _(D) ({tilde over(x)})], where I^(L) and I^(G) are segmentation label of magneticresonance and segmentation output of magnetic resonance of thegenerator, respectively.

In a step of S102, cascade-connecting the trained generator network withthe trained capsule neural network into a system to recognize andclassify the plaques in magnetic resonance imaging of vessel wall.

The trained generator network and the trained capsule neural network arecascaded into the system, which actually equalizes to the process ofmerging the training generator network with the capsule neural networkand extracting parameters from the trained generator network and thetrained capsule neural network. The merged system may realizeintegration of recognition and classification of the plaques in magneticresonance imaging of vessel wall, where the trained generator network isresponsible for recognizing the plaques from the magnetic resonanceimage of vessel wall, and the trained capsule neural network isresponsible for classifying the plaques in magnetic resonance imaging ofvessel wall.

It may be known from the example of the method of processing plaques inmagnetic resonance imaging of vessel wall shown in FIG. 1 that, in oneaspect, since the generative adversarial network includes a capsuleneural network, as compared to the traditional convolutional neuralnetwork or a fully-connected layer neural network which uses scalar torepresent the vascular plaques, the capsule neural network has moreabundant vascular plaque characteristic information represented by thevector; in another aspect, as compared to the traditional deep learningalgorithm which adopts a gradient propagation method, a capsule neuralnetwork uses the dynamic routing algorithm to learn and update thenetwork, thus, the accuracy of recognition and classification of theplaques may be greatly improved when the trained generator network andthe capsule neural network are cascaded into a system to recognize andclassify the plaques in magnetic resonance imaging of vessel wall.

FIG. 2 illustrates a schematic structural block diagram of a computerdevice 4 according to an embodiment of the present disclosure. As shownin FIG. 4, the computer device 4 in this embodiment includes a processor40, a memory 41, and a computer program 42 stored in the memory 41 andexecutable on the processor 40, such as a program of a method ofprocessing plaques in magnetic resonance imaging of vessel wall. Theprocessor 40 implement the steps in the embodiment of the method ofprocessing plaques in magnetic resonance imaging of vessel walldescribed above when executing the computer program 42, such as thesteps S101 and S102 shown in FIG. 1.

Exemplarily, computer program 42 of the method of processing plaques inmagnetic resonance imaging of vessel wall mainly includes: training agenerative adversarial network and a capsule neural network so as toobtain a trained generator network and a trained capsule neural network;cascade-connecting the trained generator network with the capsule neuralnetwork into a system to recognize and classify plaques in magneticresonance imaging of vessel wall. The computer program 42 may be dividedinto one or a plurality of modules/units, the one or plurality ofmodules/units are stored in a memory 41 and are executed by theprocessor 40 so as to implement the present disclosure. The one orplurality of modules/units may be a series of computer programinstruction segments that may accomplish particular functionalities,these instruction segments are used for describing an executive processof the computer program 42 in the computer device 4. The computer device4 may include but is not limited to: the processor 40 and a memory 41.It may be understood by the person of ordinary skill in the art that,FIG. 4 is merely an example of the computer device 4, and is notconstituted as limitation to the computing device 4, more or lesscomponents shown in FIG. 4 may be included, or some components ordifferent components may be combined; for example, the computer device 4may also include an input and output device, a network access device, abus, etc.

The so called processor 40 may be CPU (Central Processing Unit), and mayalso be other general purpose processor, DSP (Digital Signal Processor),ASIC (Application Specific Integrated Circuit), FGPA (Field-ProgrammableGate Array), or some other programmable logic devices, discrete gate ortransistor logic device, discrete hardware component, etc. The generalpurpose processor may be a microprocessor, as an alternative, theprocessor may also be any conventional processor and so on.

The memory 41 may be an internal storage unit of the computing device 4,such as a hard disk or a memory of the computer device 4. The memory 41may also be an external storage device of the computer device 4, such asa plug-in hard disk, a SMC (Smart Media Card), a SD (Secure Digital)card, a FC (Flash Card) equipped on the computer device 4. Further, thememory 41 may not only include the internal storage unit of the computerdevice 4 but also include the external storage device of the computerdevice 4. The memory 41 is configured to store the computer program, andother procedures and data needed by the computer device 4. The memory 41may also be configured to store data that has been output or being readyto be output temporarily.

In the aforesaid embodiments, the descriptions of each of theembodiments are emphasized respectively, regarding the part of oneembodiment which isn't described or disclosed in detail, reference maybe made to relevant descriptions in some other embodiments.

The person of ordinary skill in the art may be aware of that, theelements and algorithm steps of each of the examples described inconnection with the embodiments disclosed herein may be implemented inelectronic hardware, or in combination with computer software andelectronic hardware. Whether these functions are implemented by hardwareor software depends on the specific application and design constraintsof the technical solution. The skilled people could use differentmethods to implement the described functions for each particularapplication, however, such implementations should not be considered asgoing beyond the scope of the present disclosure.

It should be understood that, in the embodiments of the presentdisclosure, the disclosed device/computer device 4 and method could beimplemented in other ways. For example, the device described above aremerely illustrative; for example, the division of the units is only alogical function division, and other division could be used in theactual implementation, for example, multiple units or components couldbe combined or integrated into another system, or some features may beignored or not performed. In another aspect, the coupling or directcoupling or communicating connection shown or discussed could be anindirect coupling or a communicating connection through some interfaces,devices or units, and the coupling or direct coupling or communicatingconnection could be electrical, mechanical, or in other form.

The units described as separate components could or could not bephysically separate, the components shown as units could or could not bephysical units, which may be located in one place, or may be distributedto multiple network elements. A part or a whole of the elements could beselected according to the actual needs to achieve the objective of thepresent embodiment.

In addition, the various functional units in each of the embodiments ofthe present disclosure may be integrated into a single processing unit,or exist individually and physically, or two or more than two units areintegrated into a single unit. The aforesaid integrated unit may eitherbe achieved by hardware, or be achieved in the form of softwarefunctional units.

If the integrated unit is achieved in the form of software functionalunits, and is sold or used as an independent product, it may be storedin a computer readable storage medium. Based on this understanding, awhole or part of flow process of implementing the method in theaforesaid embodiments of the present disclosure may also be accomplishedby using computer program to instruct relevant hardware. The computerprogram of the method of processing plaques in magnetic resonanceimaging of vessel wall may be stored in a computer readable storagemedium, when the computer program is executed by the processor, thesteps in the various method embodiments described above, that is,training a generative adversarial network and a capsule neural networkso as to obtain a trained generator network and a trained capsule neuralnetwork; and cascade-connecting the trained generator network with thecapsule neural network into a system to recognize and classify plaquesin magnetic resonance imaging of vessel wall may be implemented. Where,the computer program includes computer program codes which may be in theform of source code, object code, executable documents or someintermediate form, etc. The computer readable medium may include: anyentity or device that may carry the computer program codes, recordingmedium, USB flash disk, mobile hard disk, hard disk, optical disk,computer storage device, ROM (Read-Only Memory), RAM (Random AccessMemory), electrical carrier signal, telecommunication signal andsoftware distribution medium, etc. It needs to be explained that, thecontents contained in the computer readable medium may be added orreduced appropriately according to the requirement of legislation andpatent practice in a judicial district, for example, in some judicialdistricts, according to legislation and patent practice, the computerreadable medium doesn't include electrical carrier signal andtelecommunication signal.

The aforesaid embodiments are only intended to explain but not to limitthe technical solutions of the present disclosure. Although the presentdisclosure has been explained in detail with reference to theabove-described embodiments, the person of ordinary skill in the art mayunderstand that, the technical solutions described in each of theembodiments mentioned above may still be amended, or some technicalfeatures in the technical solutions may be replaced equivalently; theseamendments or equivalent replacements, which doesn't cause the essenceof the corresponding technical solution to be broken away from thespirit and the scope of the technical solution in various embodiments ofthe present disclosure, should all be included in the protection scopeof the present disclosure.

What is claimed is:
 1. A method of processing plaques in magneticresonance imaging of vessel wall implemented by a computer devicecomprising a memory, a processor and a computer program stored in thememory and executable on the processor, the method comprising: training,by the processor of the computer device, a generative adversarialnetwork and a capsule neural network, to obtain a trained generatornetwork and a trained capsule neural network; and cascade-connecting, bythe processor of the computer device, the trained generator network withthe capsule neural network into a system to recognize and classify theplaques in magnetic resonance imaging of vessel wall; wherein thegenerative adversarial network comprises a discriminator network and agenerator network, a loss function used for training the discriminatornetwork is expressed as L(θ_(D))=E_({tilde over (x)}˜P) _(g) [D_(θ) _(D)({tilde over (x)})]−E_(x˜P) _(r) [D_(θ) _(D)(x)]+λE_({circumflex over (x)}˜P) _({circumflex over (x)})[(∥∇_({circumflex over (x)})D_(θ) _(D) ({tilde over (x)})∥₂−1)²], a lossfunction used for training the generator network is expressed asL(θ_(G))=L_(MSE)(I^(L), I^(G))+10⁻⁴L_(WANG)(I^(G)), wherein D_(θ) _(D)is a discriminator of parameter θ_(D), the x is magnetic resonanceimaging data of vessel wall, the {tilde over (x)} is data obtained afterthe magnetic resonance imaging data of vessel wall is segmented by thegenerator network, the {circumflex over (x)}=εx+(1−ε){tilde over (x)} isa random sample of random obedience distribution, the ε is random valueof uniform distribution ranged from 0 to 1, the L_(MSE)(I^(L), I^(G)) iscalculated according to a formula of${{L_{MSE}\left( {I^{L},I^{G}} \right)} = {\frac{1}{w \times h}{\sum\limits_{x = 1}^{w}{\sum\limits_{y = 1}^{h}\left( {I_{x,y}^{L} - I_{x,y}^{G}} \right)^{2}}}}},$ and the L_(WANG)(I^(G)) is calculated according to a formula ofL_(WANG)(θ_(G))=−E_({tilde over (x)}˜P) _(g) [D_(θ) _(D) ({tilde over(x)})].
 2. The method according to claim 1, wherein the step of traininga generative adversarial network and a capsule neural network so as toobtain a trained generator network and a trained capsule neural networkcomprising: taking three-dimensional local magnetic resonance imaging ofvessel wall as training data, and using Adam training algorithm anddynamic routing updating algorithm to train the generative adversarialnetwork and the capsule neural network so as to obtain the trainedgenerator network and the trained capsule neural network, by theprocessor of the computer device.
 3. A network of processing plaques inmagnetic resonance imaging of vessel wall, comprising a generativeadversarial network and a capsule neural network; wherein the generativeadversarial network is configured to recognize plaques in magneticresonance imaging of vessel wall after being trained, and the capsuleneural network is configured to classify the plaques in magneticresonance imaging of vessel wall; wherein the generative adversarialnetwork comprises a discriminator network and a generator network, thegenerator network is a deep convolutional network using a residualnetwork structure, a hybrid structure of the traditional neural networkand the capsule neural network is adopted by the discriminator network,the discriminator network or the capsule neural network comprises aconvolutional layer, a PrimaryCaps layer and a DigitCaps layer, a lossfunction used for training the discriminator is expressed asL(θ_(D))=E_({tilde over (x)}˜P) _(g) [D_(θ) _(D) ({tilde over(x)})]−E_(x˜P) _(r) [D_(θ) _(D) (x)]+λE_({circumflex over (x)}˜P)_({circumflex over (x)}) [(∥∇_({circumflex over (x)})D_(θ) _(D) ({tildeover (x)})∥₂−1)²], a loss function used for training the generatornetwork is expressed as L(θ_(G))=L_(MSE)(I^(L),I^(G))+10⁻⁴L_(WANG)(I^(G)), wherein the D_(θ) _(D) is a discriminator ofparameter θ_(D), the x is magnetic resonance imaging data of vessel the{tilde over (x)} is data obtained after the magnetic resonance imagingdata of vessel wall is segmented by the generator network, the{circumflex over (x)}=εx+(1−ε){tilde over (x)} is a random sample ofrandom obedience distribution, the ε is a random value of uniformdistribution ranged from 0 to 1, the L_(MSE)(I^(L), I^(G)) is calculatedaccording to a formula of${{L_{MSE}\left( {I^{L},I^{G}} \right)} = {\frac{1}{w \times h}{\sum\limits_{x = 1}^{w}{\sum\limits_{y = 1}^{h}\left( {I_{x,y}^{L} - I_{x,y}^{G}} \right)^{2}}}}},$ and the L_(WANG)(I^(G)) is calculated according to a formula ofL_(WANG)(θ_(G))=−E_({tilde over (x)}˜P) _(g) [D_(θ) _(D) ({tilde over(x)})].
 4. A computer device, comprising a memory, a processor and acomputer program stored in the memory and executable on the processor,wherein the processor is configured to implement steps in the methodaccording to claim 1, when executing the computer program.